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Abstract 


Prototyping  received  renewed  emphasis  in  the  Weapon  Systems  Acquisition 
Reform  Act  (WSARA)  of  2009  and  the  revision  to  DoDI  5000.02  that  preceded  it.  One 
of  the  intentions  of  WSARA  is  to  reduce  cost  growth  among  acquisition  programs.  The 
purpose  of  this  research  was  to  determine  if  cost  growth  is  reduced  for  fixed-wing  aircraft 
acquisition  programs  that  used  early  prototyping  (prior  to  MS  B).  The  research  also 
sought  to  identify  trends  between  cost  growth  and  schedule  or  prototyping  cost.  The 
research  showed  no  discernible  trends.  While  the  research  did  not  reveal  a  statistically 
significant  reduction  in  normalized  average  total  acquisition  program  cost  growth,  it  does 
offer  insight  into  the  need  to  further  clarify  acquisition  policy  guidance. 
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A  Quantitative  Analysis  of  the  Benefits  of  Prototyping  Fixed-Wing  Aircraft 


I.  Introduction 

The  DoD  has  a  long  history  of  cost  growth  associated  with  its  Major  Defense 
Acquisition  Programs  (MDAP).  The  control  of  cost  growth  in  weapon  systems  first 
gained  the  interest  of  our  nation’s  leadership  during  the  Vietnam  Conflict,  but  with 
today’s  national  budget  and  debt  crisis,  cost  growth  in  DoD  programs  has  never  been 
more  important.  Even  then  chainnan  of  the  Joint  Chiefs,  Adm.  Mike  Mullen,  has  stated 
on  numerous  occasions  that  the  single  biggest  threat  to  national  security  is  the  national 
debt  [1]  [2].  Over  the  years  several  acquisition  reform  policies  have  been  put  into  effect 
to  control  cost  growth,  all  with  varying  results  [3]  [4]  [5].  Recently,  in  an  effort  to 
combat  climbing  cost  growth,  Congress  passed  the  Weapon  Systems  Acquisition  Refonn 
Act  (WSARA)  of  2009.  Just  prior  to  its  release  a  study  was  done  by  the  Government 
Accountability  Office  (GAO)  that  reported  MDAPs  had  grown  in  cost  by  an  estimated 
$296  billion  [6], 

It  is  essential  to  institute  solid  and  sound  policy  that  can  be  clearly  followed  by 
cost  estimators,  program  managers,  and  decision  makers  alike  to  deter  cost  growth. 
WSARA  attempts  to  do  just  that.  This  research  effort  analyzes  the  wisdom  of  WSARA’s 
attempt  to  focus  on  early  systems  engineering  principles  in  the  acquisition  process, 
specifically  its  renewed  emphasis  on  competitive  prototyping  prior  to  Milestone  B 
approval. 
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General  Issue 


There  is  no  doubt  that  the  U.S.  Department  of  Defense,  along  with  the  individual 
services,  has  “historically  underestimated  the  costs  of  new  weapon  programs  [7].”  By 
underestimating  these  programs  over  the  years,  OSD  and  the  military  services  have 
underfunded  the  costs  of  these  new  weapon  systems,  leading  to  budgets  that  cannot  be 
met. 

A  study  conducted  by  RAND  on  historical  cost  growth  of  weapon  systems 
acquisition  in  2006  indicates  that  a  completed  program  will  show  an  average  cost  growth 
of  44%measured  from  Milestone  (MS)  II.  This  study  defined  a  completed  program,  as  a 
program  that  was  either  finished  or  nearly  finished  (greater  than  90%of  production 
complete)  [8].  Figure  1  below  shows  the  cost  growth  factor  for  programs  investigated  by 
RAND  (Arena,  et  ah).  The  data  is  a  distribution  of  total  cost  growth  from  Milestone  II 
and  adjusted  for  inflation  and  production  quantity  changes.  A  cost  growth  factor  less 
than  one  would  indicate  that  a  program  was  overestimated;  while  a  factor  greater  than 
one  indicates  that  a  program  was  underestimated.  The  data  show  the  mean  of  these 
programs  to  have  a  total  cost  growth  of  1 .44  based  on  adjustments  to  Milestone  II 
quantities,  demonstrating  a  clear  bias  toward  underestimating  a  program’s  cost  at  the  time 
of  Milestone  II. 

Cost  estimates  carry  with  them  a  large  degree  of  uncertainty,  and  it  is  this 
uncertainty  that  is  a  contributing  factor  toward  an  inaccurate  estimate  [9].  While 
analyzing  1 1  MDAP  programs  that  showed  significant  cost  growth;  the  Institute  for 
Defense  Analyses  (IDA)  determined  that  “weaknesses  in  initial  program  definition  and 
costing”  along  with  “weaknesses  in  management  visibility,  direction,  and  oversight”  were 
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major  causal  factors  [6].  The  IDA  report  illuminates  the  fact  that  this  uncertainty  in  the 
estimate  will  inevitably  be  visible  in  a  program’s  cost  growth  by  stating,  “If  a  program’s 
initial  cost  estimate  is  erroneously  low,  future  cost  growth  is  virtually  assured  [6].” 
WSARA  seeks  to  provide  tools  with 


CGF  range 

SOURCE:  Arena  et  al.,  2006. 

RAND  MG670-1.I 

Figure  1:  Distribution  of  total  cost  growth  from  Milestone  II  (adjusted  for  quantity  changes  and 

inflation). 

which  cost  estimators  may  combat  this  uncertainty  with  policies  emphasizing  a  renewed 
focus  in  early  systems  engineering.  One  such  policy  is  the  WSARA  2009  mandate  that 
MDAPs  use  competitive  prototyping  as  a  means  to  ensure  competition.  In  section  203 
WSARA  states: 

[T]he  acquisition  strategy  for  each  major  defense  acquisition  program  provides 
for  competitive  prototypes  before  Milestone  B  approval . . .  unless  the  Milestone 
Decision  Authority  for  such  program  waives  the  requirement. . .  [10] 

This  policy  has  been  put  into  effect  in  order  to  improve  contractor  performance 

throughout  a  program’s  life  cycle.  By  instituting  competitive  prototyping  early  in  the 
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acquisition  process,  this  policy  could  potentially  assist  cost  estimators  in  making  more 
accurate  assumptions  in  their  estimates.  If  a  competitive  prototype  does  occur  prior  to 
MS  B  will  it  affect  the  deviation  between  actual  research  and  development  and 
production  costs  from  the  cost  estimate  submitted  at  MS  B?  This  research  paper  attempts 
to  quantify  the  benefit  of  prototyping  prior  to  MS  B  by  analyzing  cost  growth  after  MS  B. 

Research  Focus 

The  Aeronautical  Systems  Center  has  sponsored  this  research  to  detennine  the 
effects  current  policy  is  having  on  the  cost  estimation  process  and  life  cycle  costs  for 
defense  acquisition  programs.  This  research  paper  will  begin  this  process  by  examining 
the  quantitative  benefits  of  prototyping  early  in  a  fixed-wing  aircraft  acquisition  program, 
and  compare  the  cost  estimates  of  these  programs  against  programs  that  did  not  have 
technology  demonstrators  or  prototypes  prior  to  MS  II  or  MS  B. 

For  this  research,  an  early  prototype  is  hardware  used  for  testing,  that  was  built 
before  full  scale  development  (FSD),  engineering  and  manufacturing  development 
(EMD),  or  system  development  and  demonstration  (SDD)  began.  The  purpose  of  the 
prototype  is  to  gain  knowledge  about  the  feasibility  of  technology,  validity  of 
requirements,  operational  suitability  of  a  concept,  and/or  resources  required  to  develop 
and  produce  a  concept.  In  summary,  the  purpose  is  the  reduction  of  risk  and  uncertainty 
for  a  future  program.  One  thing  that  should  be  kept  in  mind  is  that  not  all  prototype 
efforts  should  be  successful  in  fielding  a  weapon  system.  If  a  prototype  reveals  a 
previously  unknown  technical  problem,  invalid  requirement,  or  unaffordable  resource 
commitment  then  the  prototyping  program  was  a  success.  Anned  with  the  knowledge 
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gained  by  the  prototyping  effort,  decision  makers  can  make  an  informed  (hopefully 
better)  decision  about  the  DoD  acquisition  portfolio. 

Research  Hypothesis  and  Investigative  Questions 

This  research  started  with  expectations  to  show  lower  cost  growth  for  programs 
that  used  early  prototyping  prior  to  FSD.  The  authors  also  sought  to  find  out  if  any  cost 
growth  trends  could  be  identified  relating  to  schedule  length,  schedule  growth  and  the 
cost  of  the  prototyping  phase  in  comparison  to  the  program  costs. 

Methodology 

To  answer  the  question  on  the  benefits  of  prototyping  to  cost  estimation,  1 1  fixed- 
wing  aircraft  programs  with  either  a  prototyping  or  technology  demonstration  phase  prior 
to  commitment  to  full  scale  development  were  selected  for  analysis  based  upon  the 
preceding  definition  of  prototyping  (see  Appendix  B).  Cost  data  was  gathered  primarily 
from  the  Selected  Acquisition  Reports  (SARs)  [11]  for  each  program.  For  this  research, 
only  research,  development,  test  and  evaluation  (RDT&E),  procurement  and  military 
construction  (MILCON)  costs  were  considered.  A  program’s  operating  and  support  costs 
were  not  considered.  Initially,  the  analysis  was  intended  to  compare  the  cost  growth  of 
these  prototyped  programs  to  the  cost  growth  documented  in  [8],  This  required 
detennination  of  cost  growth  from  the  MS  B  or  MS  II  decision.  The  cost  estimate  of  the 
first  SAR  following  the  milestone  review  was  considered  to  be  the  estimate  to  compare  to 
the  costs  recorded  on  the  final  or  latest  SAR.  After  reviewing  the  list  of  prototyped 
programs,  several  were  found  to  be  amongst  those  included  in  the  study  of  comparison 
[8].  There  was  also  concern  that  only  a  small  portion  of  the  programs  included  in  the 
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contrasting  study  were  fixed-wing  aircraft  programs.  So,  although  the  prototyped 
programs’  cost  growth  data  were  compared  to  the  cost  growth  factors  (CGFs)  shown  in 
the  study  by  Arena,  et  al.  some  additional  cost  growth  factors  were  calculated  for  several 
non-pro  to  typed  fixed-wing  aircraft  programs  as  well  for  the  purposes  of  comparison  (see 
Appendix  C).  Cost  growth  factors  were  normalized  for  quantity  changes  and  inflation. 

Assumptions/Limitations 

The  time  allotted  to  complete  this  study  did  not  allow  a  thorough  search  for 
sources  of  cost  estimates  and  actual  costs  incurred  from  other  than  the  SARs.  Therefore, 
programs  that  pre-date  the  publishing  of  SARs  (1969)  could  not  be  included  in  the 
sample. 

In  addition  to  comparing  the  MS  B  or  MS  II  cost  growth  to  the  cost  growth 
documented  by  Arena,  et  al.,  there  was  also  a  desire  to  compare  the  cost  growth  from  the 
time  of  the  Preliminary  Design  Review  (PDR)  to  the  cost  growth  from  the  MS  review. 
However,  it  was  found  that  determining  the  date  of  PDR  was  not  possible  for  many 
programs.  For  the  programs  that  did  have  PDR  dates  available  many  of  the  dates  were 
within  months  of  the  MS  decision  so  there  was  not  a  separate  cost  estimate  associated 
with  PDR.  Ultimately  only  three  programs  had  a  PDR  date  that  was  confirmed  to  be  at 
least  a  year  different  from  the  date  that  the  program  met  with  the  milestone  decision 
authority  (MDA)  for  a  MS  II  or  MS  B  decision. 
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II.  Literature  Review 


Cost  Growth 

Cost  growth  in  weapon  system  acquisition  has  been  a  consistent  critique  of  the 
Department  of  Defense  and  the  Services.  Disappointed  with  continued  poor  perfonnance 
with  respect  to  acquisition  program  cost  and  schedule,  Congress  enacted  the  WSARA  of 
2009  [12].  WSARA  includes  provisions  to  ensure  programs  are  based  on  realistic 
estimates  and  that  programs  can  be  cancelled  for  high  levels  of  cost  growth.  However, 
cost  growth  in  weapon  systems  procurement  has  a  long  history  that  predates  WSARA 
and  instigated  the  original  DoD  5000  Directive  and  Instruction  in  1971.  Many  studies 
have  sought  to  describe  trends  and  identify  the  sources  of  weapon  system  cost  growth; 
the  findings  are  summarized  below. 

The  basic  trend  in  cost  growth  can  be  summed  up  with  one  word  -  consistency. 
Studies  that  looked  for  trends  across  acquisition  programs  and  within  acquisition 
programs  since  the  inception  of  the  DoD  5000  have  shown  no  real  improvement  in  cost 
growth.  In  fact,  research  shows  that  “cost  growth  was  higher  during  the  late  1960s,  lower 
in  the  early  1970s  and  higher  in  the  late  1970s  [4].”  Drezner,  et  al.  came  to  the 
conclusion  that  “no  substantial  improvement  in  average  cost  growth”  had  occurred  over 
the  three  decades  leading  up  to  1993  [13].  In  a  more  recent  study,  the  authors  found  that 
“development  cost  growth  in  the  past  three  decades  has  remained  high,  with  no 
significant  improvement  [14].”  But,  they  did  find  greater  variability  in  development  cost 
growth  during  the  1990s. 

In  addition  to  cost  trends  across  programs,  Arena,  et  al.  examined  the  cost  growth 
trend  over  time  within  programs.  As  expected,  they  found  the  CGF  tends  toward  1.0  and 
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the  variability  decreases  as  a  program  matures  [8],  However,  the  authors  also  showed 
that  the  CGF  does  not  settle  to  an  average  of  1  for  the  programs  considered  until  70-80% 
of  each  program  had  been  executed  from  the  time  of  MS  II  to  the  end  of  the  program  as 
indicated  on  the  final  SAR.  Younossi,  et  al.  found  that  development  cost  growth 
increased  sharply  up  to  about  40%  of  the  time  lapse  between  MS  B  and  the  costs 
published  in  the  final  SAR  followed  by  shallower  cost  growth  throughout  the  remainder 
of  the  program  to  include  the  procurement  phase  [14].  The  same  study  revealed  that 
procurement  cost  growth  was  steepest  for  programs  up  to  approximately  40%t  of  the 
actual  schedule  between  the  MS  B  date  and  the  date  of  the  final  SAR. 

The  actual  cost  growth  factors  reported  in  the  various  studies  are  not  always 
consistent  even  though  the  studies  consistently  show  a  cost  growth  phenomenon.  The 
reason  for  the  differences  seems  to  be  related  to  a  few  factors:  (1)  some  studies  only 
include  completed  acquisition  programs  which  have  recognized  all  their  cost  growth  in 
contrast  to  studies  that  included  on-going  programs,  (2)  differences  in  how  the  quantity 
normalization  was  accomplished,  and  (3)  differences  in  the  type  of  programs  examined 
(e.g.,  space  systems,  aircraft,  helicopters,  etc.  [8]).  In  [13],  the  weighted  average  total 
program  cost  growth  was  shown  to  be  about  20%t  from  both  the  planning  and 
development  baselines.  A  decade  later,  McNicol  found  that  30%  of  the  major  systems  in 
the  DoD  acquired  between  1970  and  1997  showed  negative  cost  growth  (the  cost  had 
been  over  estimated)  [15].  Further,  of  138  systems  examined,  about  80%had  a  cost 
growth  less  than  30%.  Approximately  20%of  the  systems  had  a  cost  growth  of  more  than 
30%.  The  authors  of  [8]  found  an  average  adjusted  total  cost  growth  for  a  completed 
acquisition  program  was  46%  when  measured  from  the  MS  II  estimate.  Arena,  et  al.  also 
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found  that  procurement  cost  growth  was  44%and  development  cost  growth  was  almost 
60%.  Finally,  they  showed  that  most  of  the  average  cost  growth  was  due  to  relatively 
few  systems  that  experienced  extreme  cost  growth  just  as  McNicol  had  shown 
previously.  Table  2.1  in  [8]  provides  an  extensive  overview  of  cost  growth  factors  from 
studies  examining  cost  growth  prior  to  2006. 

As  was  mentioned  earlier,  the  purpose  of  many  studies  was  to  identify  the  causes 
and  consequences  of  cost  growth.  One  of  the  most  consistently  cited  reasons  for 
increased  cost  in  acquisitions  is  program  schedule  growth  or  extended  programs  [4]  [14] 
[15]  [16]  [17]  [18].  Reference  [13]  showed  mixed  results  with  regard  to  schedule. 
Drezner  found  no  correlation  between  schedule  slip  and  cost  growth,  but  did  find  that 
longer  programs  have  greater  cost  growth.  In  [4],  Tyson,  et  al.  stated  that  production 
stretch  increased  cost  growth  by  7  to  10%per  unit  increase  in  schedule  stretch.  Bolton,  et 
al.  attributed  9%of  cost  growth,  on  average,  to  schedule  changes  [18]. 

Another  instigator  of  cost  growth  that  receives  much  scrutiny  is  overly  optimistic 
cost  estimates  early  in  a  program.  Several  studies  have  claimed  that  cost  estimates  were 
often  unrealistic  and  that  independent  cost  estimates  are  needed  to  combat  cost  growth 
[15]  [16]  [18]  [6]  [17]  [19].  Bolton,  et  al.  cited  errors  in  cost  estimation  to  account  for 
10%of  cost  growth  experienced  by  35  MDAPs  [18].  In  [6],  Porter,  et  al.  mentioned  that 
highly  optimistic  assumptions  about  the  cost-saving  benefits  of  acquisition  reform 
initiatives  were  to  blame  for  some  cost  estimation  errors.  They  also  pointed  out  that 
program  offices  and  the  services  have  a  detrimental  tendency  to  match  their  cost 
estimates  to  the  expected  funding  available  for  a  program.  In  [5],  Lorell  and  Graser 
specifically  addressed  acquisition  refonn  which  will  be  touched  upon  later.  O’Neil 
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introduces  the  tenn  “planning  fallacy”  to  account  for  the  overly  optimistic  estimates  of 
what  a  program  can  accomplish.  Planning  fallacy  refers  to  the  “pervasive  human 
tendency  to  hold  ‘the  conviction  that  a  current  project  will  go  as  well  as  planned  even 
though  most  projects  from  a  relevant  comparison  set  have  failed  to  fulfill  their  planned 
outcomes.’”  Planning  fallacy  often  occurs  at  multiple  layers  of  decision  making  within 
the  acquisition  system  [19].  Along  with  Porter,  et  al.  [6],  O’Neil  suggested  that  an  over 
reliance  on  contractor  cost  estimates  can  lead  to  unrealistically  optimistic  initial  program 
estimates.  He  asserted  that  contractors  often  promise  too  much  because  the  “incentives 
weigh  heavily  on  the  side  of  accepting  future  risks  rather  than  immediate  ones.”  In  this 
case,  the  contractor  can  risk  failure  to  be  selected  for  a  program  by  providing  a  more 
realistic  view  of  the  resources  required,  which  could  lead  to  corporate  suicide,  or  fight  to 
be  selected  for  a  problematic  program  that  will  keep  the  company  in  business.  The  errors 
in  cost  estimation  impacted  programs  with  an  effective  2  to  8%annual  tax  on 
procurement  according  to  McNicol  [15]. 

A  few  studies  categorized  the  causes  of  cost  growth.  McNicol  used  decisions  and 
mistakes  to  bin  cost  growth.  He  found  that  decisions  accounted  for  21  and  10%of  cost 
growth  from  MS  II  during  development  and  procurement,  respectively,  and  mistakes 
were  attributed  with  24  and  18%of  cost  growth  for  development  and  procurement, 
respectively  [15].  Similarly,  Arena,  et  al.  considered  economy-driven  and  customer- 
driven  variables  to  classify  cost  growth  sources  [20].  This  study  found  that  labor, 
material,  equipment  and  manufacturing  helped  increase  the  cost  of  fixed-wing  aircraft 
about  3.5%annually  and  accounted  for  one-third  of  cost  escalation.  Customer-driven 
factors  accounted  for  two -thirds  of  the  contribution  to  cost  escalation.  Production  rate 
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and  technical  complexity  were  chief  drivers  of  customer-driven  factors.  Finally,  Bolton, 
et  al.  grouped  SAR  variances  into  the  following:  (1)  errors  in  estimation  and  planning, 

(2)  decisions  by  the  government,  (3)  financial  matters,  and  (4)  miscellaneous  sources 
[18].  The  authors  found  total  cost  growth  to  be  dominated  by  decisions,  which  accounted 
for  more  than  two-thirds  of  the  growth.  Interestingly,  the  decision  category  was  similar 
to  the  customer-driven  category  defined  by  Arena,  et  al.  and  the  proportions  of  cost 
growth  were  nearly  equal.  The  primary  factors  within  the  decision  category  were 
quantity  changes  (22%),  requirements  growth  (13%)  and  schedule  changes  (9%).  As 
mentioned  previously,  cost  estimation  was  blamed  for  10%of  cost  growth  in  the  error 
category. 

Lorell  and  Graser  suggested  three  types  of  acquisition  reform::  reducing 
regulations  and  oversight,  using  commercial-like  program  structures,  and  multiyear 
procurement.  They  found  that  only  the  state  of  regulation  and  oversight  following  the 
Federal  Acquisition  Streamlining  Act  of  1994  correlated  to  program  savings  (on  the  order 
of  3  to  4%)  [5],  They  also  stated  that  multiyear  contracts  may  save  about  5  to  10%on 
production  contracts,  which  agrees  with  the  findings  of  Tyson,  et  al.  [4].  Lorell  and 
Graser  were  unable  to  identify  cost  estimating  relationships  for  any  acquisition  reform 
cost  savings.  They  also  felt  there  were  too  many  caveats  for  the  use  of  commercial-like 
practices  to  make  a  definitive  statement  about  its  benefits. 

Another  source  of  cost  growth  regularly  reported  in  the  literature  is  requirements 
growth  or  changes,  unrealistic  requirements  (or  at  least  unexamined  for  cost 
effectiveness)  and  requirements  that  are  not  matched  to  available  resources  [15]  [18]  [6] 
[20]  [21]  [22]  [23]. 
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One  strategy  to  address  cost  growth  has  received  mixed  reviews  by  previous 
studies — prototyping.  Tyson,  et  al.  found  prototyping  before  full  scale  development  had 
generally  been  successful  [4].  A  few  years  later  Drezner,  et  al.  showed  that  prototypes 
are  associated  with  higher  cost  growth  [13].  They  did  show  that  programs  that  used 
prototyping  prior  to  starting  EMD  had  lower  cost  growth  than  programs  that  prototype 
post-EMD.  These  programs  were  also  two  years  shorter  in  duration  on  average  than  the 
post-EMD  prototyped  programs.  However,  they  showed  that  procurement  cost  growth 
for  aircraft  was  the  same  for  both  prototyping  and  non-prototyping  programs.  These 
results  are  interesting  as  one  would  expect  early  prototyping  to  increase  the  knowledge  of 
the  feasibility  of  a  program  yielding  consistently  lower  cost  growth. 

Prototyping 

The  Defense  Acquisition  University  defines  a  prototype  as  “an  original  or  model 
on  which  a  later  system/item  is  fonned  or  based  [24].”  Over  the  years,  many  studies  have 
been  conducted  on  the  usefulness  of  prototyping  in  major  weapon  system  acquisition 
programs.  Specifically,  the  belief  is  that  prototyping  can  be  used  as  a  tool  in  order  to 
“obtain  better  infonnation  for  EMD  decisions  and  technical/schedule/cost  risk  reduction 
[25].” 

Recently  Congress  and  the  DoD  have  introduced  legislation  that  reforms  the 
Defense  Acquisition  process.  Two  of  their  goals  are  to  “improve  the  department’s  ability 
to  balance  requirements  with  resources  and  establish  a  stronger  foundation  for  starting 
programs  [26].”  DoDI  5000.02  adds  detail  to  WSARA’s  mandate  of  prototyping  early  in 
a  program: 
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The  [Technology  Development  Strategy]  and  associated  funding  shall  provide  for 
two  or  more  competing  teams  producing  prototypes  of  the  system  and/or  key 
system  elements  prior  to,  or  through,  Milestone  B.  Prototype  systems  or 
appropriate  component-level  prototyping  shall  be  employed  to  reduce  technical 
risk,  validate  designs  and  cost  estimates,  evaluate  manufacturing  processes,  and 
refine  requirements.  [27] 

A  prototype  phase  within  a  program  prior  to  EMD  can  prove  essential  to  achieving  both 
goals,  and  may  also  prove  necessary  in  discovering  “if  the  chosen  concept  can  be 
developed  and  produced  within  existing  resources  [26].” 

Prototypes  can  give  the  decision  maker  insight  into  technical  risks  and  uncertainty 
that  may  arise  within  a  program  early  on,  so  that  steps  may  be  taken  to  correct  these 
issues  before  cost  or  schedule  overruns  occur.  Additionally,  competitive  prototypes  can 
be  a  means  by  which  competition  is  extended  deeper  into  the  development  cycle,  while 
keeping  costs  under  budget.  This  can  lead  to  a  more  competitive  contract  environment. 
Ultimately,  the  goal  of  prototyping  is  to  successfully  resolve  technical  risk  as  early  in  a 
program  as  possible  in  order  to  reduce  cost. 

In  1991  IDA  conducted  a  comparison  of  defense  system  programs.  The  study 
identified  total  development  cost  growth  of  prototyped  programs  was  17%  compared  to 
62%  for  those  without  a  prototype.  Results  were  similar  for  production  cost  growth. 
Programs  with  prototypes  demonstrated  a  29%  increase  in  production  cost  while  those 
without  ballooned  to  55%  [28], 

Ten  years  earlier  The  RAND  Corporation  conducted  a  study  between  four 
airborne  acquisition  programs:  the  A- 10,  F-16,  UH-60,  and  AH-64.  This  report 
identifies  prototyping  as  a  “risk  reduction  strategy,”  adding  “if  the  new  system  involves 
much  technical  risk,  a  prototype  phase  can  probably  improve  the  efficiency  of  finding 
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and  correcting  many  of  the  inevitable  design  flaws  [29].”  The  difficulty  lies  with 
correctly  identifying  the  technical  risk.  This  was  an  essential  aspect  to  why  the  Light 
Weight  Fighter  program  was  prototyped.  It  was  during  the  prototype  phase  of  the  Light 
Weight  Fighter  (later  known  as  the  F-16)  that  “the  fly-by- wire  control  and 
autostabilization  system  was  refined  and  proven  to  work  [28].”  If,  however,  the  wrong 
risk  is  identified  as  a  lynchpin  risk  then  the  incorrect  system  may  be  prototyped.  This 
could  have  disastrous  results. 

If  the  wrong  system  is  prototyped  then  the  program  could  be  set  back  months  or 
years  behind  schedule.  Additionally,  vast  amounts  of  money  will  be  wasted,  and  the 
program  decision  maker  could  be  making  risk  based  decisions  on  a  prototype  that  at  best 
doesn’t  matter  or  at  worse  provides  a  false  sense  of  knowledge  security.  This  can  lead  to 
improper  cost  estimates  that  inevitably  challenge  a  program.  If,  however,  the  correct 
system  is  identified  and  prototyped  then  decision  makers  could  see  a  return  on  their 
investment  in  the  form  of  cost  growth  control.  “There  are  at  least  three  ways  in  which  the 
existence  of  a  prototype  phase  might  improve  the  ability  to  estimate  future  program  costs, 
or  to  control  subsequent  cost  growth:  an  improved  database  for  estimation,  fewer 
unexpected  configuration  changes,  and  use  of  fixed  price  contracts  [29].” 

A  solid  prototyping  program  can  provide  a  database  from  which  cost  estimators 
may  pull  current  and  realistic  data  by  which  to  more  accurately  estimate  program  costs. 
Traditional  cost  estimates  are  made  by  comparing  similar  programs  and  taking  previous 
programs’  costs  to  use  as  baseline  estimates.  For  early  estimations  this  will  suffice,  but 
as  programs  move  forward  more  detailed  program  estimates  become  possible  and  critical. 
It  is  challenging,  however,  to  find  previous  programs  to  act  as  baselines  if  the  new 
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program  contains  unique  technical  risks.  Some  analysts  will  argue  that  having  an  actual 
flying  piece  of  hardware  can  bridge  the  gap  between  “similar”  previous  programs  and  a 
current  program’s  estimate.  “In  the  Light  Weight  Fighter  program  General  Dynamics 
used  data  from  their  prototypes  as  a  basis  for  cost  estimates  for  the  full  scale  development 
and  production  phases.”  Their  prototypes  gave  “General  Dynamics  management  an 
estimate  of  the  labor  hours,  materiel  costs  and  tooling  that  would  be  required  to  build 
their  aircraft  model  using  production  line  methods  [29].” 

A  significant  source  of  cost  growth  is  the  necessity  to  reconfigure  a  program  in 
order  to  account  for  unexpected  technical  difficulties.  “A  prototype  evaluation  before 
FSD  might  reduce  the  number  of  corrective  design  revisions  [29]”  that  can  plague  most 
development  programs.  The  assumption  here  is  that  a  flying  test  prototype  will  highlight 
many  of  the  design  changes  that  are  necessary  early  on  in  a  program.  Discovering  these 
changes  earlier  rather  than  later  allows  the  developer  to  incorporate  these  changes  into 
early  full-scale  development  models,  and  avoid  the  growth  in  cost  that  would  be  caused 
by  retrofitting  these  changes  in  later  blocks.  Additionally,  “the  FSD  phase  following  a 
prototype  phase  is  considerably  shorter,  on  average,  than  it  is  in  those  programs  that 
omitted  a  prototype  phase  [29].”  Based  on  their  analysis  of  the  four  programs  in  the 
1970s,  the  report  “suggests  that  the  inclusion  of  a  prototype  phase  does  not  usually 
extend  the  total  development  time  [29].” 

These  previously  studied  articles  would  lead  one  to  believe  that  prototyping  is  the 
answer  to  all  of  a  cost  estimator’s  problems.  A  perfect  or  one  size  fits  all  solution,  but 
there  are  other  articles  that  refute  these  claims.  Stating,  in  actuality  that  “there  are  few 
significant  differences  between  prototyping  and  nonprototyping  programs  with  respect  to 
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cost  growth,  total  actual  program  duration,  or  schedule  slip  [30].”  Drezner  goes  on  to 
claim  that  although  “dollar-weighted  average  cost  growth  ratio  for  the  prototyping 
programs  is  1.30  while  nonprototyping  programs  average  1.1 6... The  evidence  linking 
prototyping  and  program  outcomes  is  fairly  weak  [30].”  The  author  goes  on  to  state  that 
after  studying  90  major  weapon  system  acquisition  programs  “most  of  the  differences 
between  prototyping  and  nonprototyping  programs  were  not  significant  from  a  statistical 
standpoint.”  This  study  highlights  the  importance  of  properly  identifying  a  prototyping 
strategy.  One  that  focuses  less  on  the  need  to  prototype,  and  more  on  identifying  the 
technology  challenges  and  high  risk  areas  that  should  be  prototyped.  In  the  end  a 
successful  program  may  result  more  so  because  of  this  critical  identification  rather  than 
just  because  the  program  was  prototyped.  This  is  very  difficult  to  prove  because  it  is  so 
ambiguous.  “We  cannot  know  what  the  outcome  of  a  prototyping  program  would  have 
been  if  it  was  not  prototyped;  we  can  only  speculate  [30].”  It  is  because  of  this  difficulty 
that  this  study  suggests  that  policy  remain  flexible  with  respect  to  prototyping. 

So,  does  prototyping  help  cost  estimation,  or  is  it  a  waste  of  money?  Perhaps  the 
answer  rests  more  firmly  with  specifically  what  is  prototyped  rather  than  whether 
prototyping  occurred.  Program  decision  makers  must  decide  whether  a  prototype  is 
appropriate  and  cost  effective.  Clearly  programs  with  unique  technological  risks 
involved  could  stand  to  benefit  from  a  prototype  phase  prior  to  FSD,  especially  if  a 
program  is  attempting  to  leverage  immature  technologies.  In  the  1970s,  the  HAVE 
BLUE  program  “demonstrated  that  a  manned  aircraft  could  achieve  radar  signatures  low 
enough  to  conceivably  perfonn  tactical  missions  without  being  detected  [31]  [3 1].”  The 
success  of  this  prototype  is  not  only  measured  within  the  F-l  17  program  but  throughout 
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descendant  programs  within  the  DoD,  making  the  measurement  of  success  less 
quantitative  and  more  qualitative.  This  demonstration  allowed  the  program  to  move 
forward  under  a  strong  foundation  that  the  technology  was  ready  and  could  achieve  the 
requirements  of  the  program.  Additionally,  the  Advanced  Medium  STOL  Transport 
(AMST)  program  provided  “a  wealth  of  practical  experience  and  engineering  data  on 
large  [short  takeoff  and  landing]  (STOL)  tactical  transports  and  operational 
employment.”  It  also  “gave  insight  into  the  costs  of  developing  such  an  aircraft  for 
production  in  the  event  that  a  program  should  be  pursued  in  the  future  [32].”  It  was 
because  of  the  fact  that  the  AMST  program  honed  in  on  the  STOL  requirement  that 
allowed  the  prototypes  to  focus  on  the  aspects  of  the  technology  that  would  allow  the 
program  to  be  successful.  In  this  case  the  correct  technology  limitations  were  both 
identified  and  prototyped.  Both  prototypes  focused  on  the  engines  and  powered  lift 
design.  This  would  ultimately  enable  the  successful  achievement  of  the  STOL 
requirement,  and  make  the  “AMST  program  a  vital  step  in  developing  the  technology  that 
made  the  C-17  Globemaster  III  possible  [32].” 

Historically,  emphasis  on  prototyping  has  “waxed  and  waned  throughout  the  Air 
Force  [33].”  When  selecting  the  best  strategy  for  moving  forward  with  a  major  weapon 
system  acquisition,  program  leadership  will  debate  the  benefits  of  a  prototype.  There  is 
much  literature  available  to  add  guidance  on  the  merits  of  prototyping,  but  depending  on 
what  is  researched  the  results  seem  to  vary.  The  literature  studied  for  this  project; 
however,  all  seem  to  agree  that  if  a  program  is  to  move  forward  with  a  prototype  phase 
prior  to  FSD  understanding  what  needs  to  be  prototyped  is  critical.  The  challenge  for 
program  managers  is  to  “clearly  focus  on  end  product  objectives,  and  then  conduct 
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technology  demonstrations  that  provide  the  essential  information  for  these  systems  [33].” 
Ultimately  this  is  the  driving  force  behind  a  program’s  success  or  failure.  None  of  the 
literature  found  has  tried  to  address  the  issue  of  prototyping  in  the  last  decade,  when 
systems  have  become  increasingly  complex.  Much  of  the  literature  has  based  their 
results  on  limited  numbers  of  programs  during  a  similar  time  frame,  or  on  large  numbers 
of  data  pulled  from  multiple  major  weapon  system  acquisition  programs  throughout  the 
DoD  that  have  few  similarities.  They  all  seem  to  agree  that  there  might  be  an  ambiguous 
benefit  to  prototyping,  but  based  on  these  readings  it  is  difficult  to  quantify  exactly  what 
should  be  prototyped,  or  what  if  any  influence  prototyping  ultimately  has  on  the  success 
or  failure  of  an  acquisition  program. 

Summary 

As  stated,  “cost  growth  acts  like  a  tax,  squeezing  all  acquisition  programs  and 
causing  inefficiencies  [17].”  Studies  have  shown  that  cost  growth  is  a  persistent  problem 
for  the  DoD  that  spans  decades.  Poor  estimates,  requirements  instability,  program  delays 
and  numerous  other  causes  have  been  cited  for  cost  growth.  Ultimately,  the  blame  for 
most  cost  growth  can  attributed  to  poor  decision  making,  particularly  at  the  inception  of 
many  programs.  Generally,  decisions  are  only  as  good  as  the  knowledge  available  to 
support  the  decision.  Some  programs  have  no  doubt  been  initiated  against  better 
judgment  or  in  disregard  of  the  gaps  in  knowledge  about  the  magnitude  of  the  challenge 
presented  by  a  program.  WSARA  and  DoDI  5000.02  have  placed  renewed  emphasis  on 
prototyping  in  the  hopes  that  it  will  support  knowledge  based  decision  making,  and 
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ultimately  resolve  technical  risk  as  early  in  a  program  as  possible  in  order  to  reduce  costs 
or  at  least  cost  growth. 
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III.  Methodology 


Cost  growth  was  determined  based  on  cost  estimates  and  actual  costs  reported  in  a 
program’s  SARs.  Costs  reported  in  each  program’s  SAR  associated  with  MS  II  (or  MS 
B)  were  gathered  and  then  compared  to  the  base-year  cost  data  found  in  the  program’s 
final  or  latest  SAR.  In  a  few  cases,  prior  to  the  mid-1970s,  cost  data  available  in  the 
SARs  were  only  presented  as  then-year  dollars.  Also,  most  of  the  more  recent  programs 
experienced  base-year  changes  between  the  MS  II  or  B  estimate  and  the  final  or  latest 
SAR.  For  these  instances,  the  data  were  inflated  from  then- year  dollars  or  from  the  base- 
year  on  the  SAR  associated  with  the  MS  decision  to  the  base-year  used  on  the  final  or 
latest  SAR.  Then-year  dollars  were  converted  by  using  the  weighted  indices,  and  the 
base-year  dollars  were  inflated  by  using  the  raw  indices  of  the  OSD  inflation  tables  [34] 
[34]  [34]  [35]  [35].  This  approach  removes  the  majority  of  inflation  related  cost  growth, 
which  is  common  practice  in  studying  cost  growth. 

Once  the  results  of  inflation  had  been  removed  from  the  data  there  was  still  an 
aspect  of  growth  related  to  procurement  quantity  changes  between  the  MS  II  or  B 
estimate  and  the  final  or  latest  SAR.  To  remove  the  effects  of  quantity  changes,  all  the 
SAR  procurement  cost  estimates  were  adjusted  to  the  MS  B  or  II  quantity  estimate.  The 
adjustment  uses  a  cumulative  average  cost  improvement  curve  (CIC)  derived  from  annual 
funding  data  provided  in  each  program’s  SAR.  The  recurring  and  non-recurring  costs 
must  be  separated  for  each  year  of  funding,  accumulated  year  by  year  and  then  plotted 
against  the  cumulative  quantity  procured  for  each  year  to  develop  the  cumulative  CIC 
(also  known  as  a  learning  curve).  This  methodology  is  described  by  Hough  in  [36]  and  in 
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greater  detail  by  Dews,  et  al.  in  Appendix  A  of  [37].  If  the  production  estimate  at  the 
milestone  review  was  greater  than  the  number  of  aircraft  actually  built  then  the  cost  of 
purchasing  the  un-built  aircraft  that  were  included  in  the  original  estimate  were  added  to 
the  procurement  cost  of  the  program.  Likewise,  if  the  production  estimate  was  lower 
than  the  actual  production  run  then  the  cost  of  the  additional  aircraft  were  subtracted  from 
the  final  cost  of  the  program.  This  methodology  differs  slightly  from  what  was  used  in 
the  comparison  study  because  Arena,  et  al.  normalized  the  cost  data  to  the  final  quantity 
produced.  However,  Appendix  C  of  [8]  presented  the  data  for  nonnalization  to  the  MS  II 
quantity.  So,  a  direct  comparison  of  CGFs  is  possible  between  the  two  studies. 

Figure  2  shows  the  amount  of  total  cost  removed  or  added  (as  in  the  case  of  the 
example  in  the  figure)  to  the  current  estimate  (CE)  based  upon  the  difference  between  the 
quantity  planned  at  the  time  of  the  development  estimate  (DE)  and  the  quantity  actually 
procured  or  planned  at  the  time  of  the  CE.  This  quantity  nonnalization  method  provides 
a  consistent  basis  for  normalization  as  the  planned  quantity  at  MS  II  or  B  does  not 
change,  unlike  the  quantity  of  the  current  estimate  which  can  change  from  SAR  to  SAR. 
For  the  example  below,  if  the  program  had  procured  the  number  of  aircraft  that  were 
planned  at  the  time  of  MS  B  then  the  total  program  cost  would  have  increased 
significantly  over  the  cost  reported  in  the  CE.  So,  calculating  a  CGF  based  on  the  DE 
created  under  the  assumption  of  a  large  total  quantity  and  the  CE  based  on  a  much  lower 
actual  total  quantity  is  distorted,  appearing  to  be  much  lower  than  what  an  honest 
assessment  of  the  program  cost  would  show. 
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Figure  2:  Quantity  normalization  is  accomplished  by  removing  the  difference  between  the  cost 
estimate  at  the  DE  quantity  and  the  CE  quantity  on  the  total  cost  curve  (recurring  plus  non-recurring 

cost)  for  the  current  estimate. 

A  few  of  the  older  programs  used  in  the  study  had  ‘non  standard’  cost  data 
available  (by  current  SAR  standards)  when  compared  to  the  other  programs  used  in  the 
study.  The  first  of  these  programs  was  the  A- 10.  The  data  available  for  the  A- 10  did  not 
allow  the  separation  of  the  recurring  and  non-recurring  costs.  Therefore,  a  CIC  could  not 
be  determined.  In  the  case  of  the  A- 10,  the  final  procurement  quantity  was  727  aircraft, 
and  the  MS  II  planned  quantity  was  729.  Although  this  is  a  small  change,  the  quantity 
cost  variance  contained  in  the  final  SAR  was  used  to  normalize  for  the  quantity  change. 
This  same  problem  arose  for  some  of  the  programs  used  as  comparison  non-prototyping 
programs,  and  these  cases  were  resolved  in  the  same  manner  (see  Appendices  B  and  C). 
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The  F- 117  program  also  presented  some  difficulty  in  detennining  a  CIC.  The 
data  obtained  from  SPO  documentation  did  not  provide  a  year  by  year  breakdown  of 
RDT&E  costs.  However,  recurring  and  non-recurring  costs  for  procurement  were 
available  by  fiscal  year  in  then-year  dollars.  The  RDT&E  costs  through  FSD  were 
provided  in  then- year  dollars  as  a  lump  sum.  Additionally,  the  cost  of  full  capability 
development  was  available  in  then-year  dollars  as  a  lump  sum.  Full  capability 
development  was  the  RDT&E  that  continued  after  the  completion  of  the  FSD  contract, 
which  ended  in  July  1984.  In  [3 1]  [31],  the  authors  stated  that  full  capability 
development  occurred  through  approximately  1990.  Procurement  did  not  start  until 
FY80.  So,  for  this  study  the  RDT&E  costs  were  distributed  from  FY79  through  FY90 
according  to  power  curves  fit  to  the  first  5  years  and  the  following  6  years  of  RDT&E 
annual  cost  data  from  the  A- 10,  F-16  and  F-18  programs  combined,  which  occurred 
during  a  similar  time  period  and  are  all  fighter/attack  aircraft  programs.  As  noted  in  [3 1] 
[31],  the  RDT&E  spending  profile  for  the  F- 117  was  a  bit  unusual  due  to  the  structure  of 
the  program  and  the  amount  of  development  that  occurred  after  the  FSD  contract 
completed.  Once  the  RDT&E  spend  profile  was  detennined,  the  CIC  was  calculated  as 
discussed  previously. 

After  calculating  the  average  CGF  for  the  sample  programs  a  test  of  statistical 
significance  was  conducted  on  the  mean  as  discussed  in  [38].  The  significance  testing 
was  accomplished  for  the  log  nonnal  distribution  discovered  by  Arena,  et  al.  in  their 
study.  Based  on  the  findings  of  Arena,  et  al.  the  distributions  of  the  data  gathered  for  this 
study  were  assumed  to  be  log  nonnal  as  well.  Using  the  equations  for  the  log  normal 
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mean  and  variance  defined  in  [39,  p.  625]  the  mean  and  variance  for  log  space  were 
calculated  for  the  data  presented  in  Tables  C.  1  and  C.2  of  the  comparison  RAND  study. 
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IV.  Analysis  and  Results 


Overview 

In  this  chapter,  the  authors  analyze  the  CGF  and  SAR  data  of  1 1  programs  that 
contained  a  prototype  phase  prior  to  FSD.  This  data  is  then  compared  with  the  cost 
growth  data  collected  by  Arena,  et  al  in  [8]  to  identify  what,  if  any,  influence  prototypes 
have  on  cost  growth  in  an  acquisition  program. 


Results 

Initial  Findings 

SAR  cost  data  is  typically  presented  under  the  following  categories:  development 
(RDT&E),  procurement,  military  construction  (MILCON)  and  total  cost.  Additionally, 
there  are  operations  and  maintenance  costs  related  only  to  the  acquisition  portion  of  the 
program.  As  described  in  the  previous  chapter,  the  authors  calculated  CGFs  adjusted  for 
quantity  change  based  on  MS  II  (or  B)  SAR  quantity  estimates.  Table  1  is  a  summary  of 
the  statistical  totals  for  all  1 1  programs  studied. 


Table  1:  CGF  Summary  Statistics  by  Funding  Categories  for  MS  II 


Category 

Number  of 
Programs 

Mean 

Median 

Standard 

Deviation 

Minimum 

Maximum 

Total 

11 

1.52 

1.42 

0.43 

0.97 

2.40 

RDT&E 

11 

2.15 

1.54 

1.15 

1.32 

5.04 

Procurement 

11 

1.43 

1.36 

0.39 

0.89 

2.03 

The  table  shows  the  mean  total  CGF  to  be  greater  than  that  of  1 .44  established  by 


the  comparison  study  [8].  At  first  glance  there  does  not  seem  to  be  a  relationship 


between  reduced  cost  growth  and  prototyping.  An  important  observation,  however,  is 


that  the  data  appears  to  be  skewed  because  the  mean  is  higher  than  the  median  for  all 
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categories.  One  of  the  reasons  for  this  skew  is  the  performance  of  the  UAV  programs.  As 
an  example  the  Predator  program  performed  very  poorly  with  an  RDT&E  CGF  of  5.04 
and  a  total  program  CGF  of  2. 12.  This  could  be  attributed  to  the  fact  that  the  Predator 
RDT&E  spending  profile  was  opposite  that  of  most  of  the  other  programs.  This  program 
had  more  of  a  building  block  approach  to  RDT&E  spending,  with  almost  50%  of  the  total 
cost  of  RDT&E  occurring  in  the  last  three  years  of  a  13  year  schedule.  Figure  3  shows 
the  distribution  of  the  total  CGF  for  the  1 1  programs  studied.  The  distribution  portrays 
the  existence  of  two  outlier  programs,  in  this  case  Predator  and  Global  Hawk,  with  total 
CGFs  of  2.12  and  2.40  respectively. 

5 - 

4 - 


CGF  Range 

Figure  3:  Distribution  of  Total  Cost  Growth  from  MS  II  Adjusted  for  Quantity  Changes 

In  order  to  better  evaluate  the  performance  of  this  data,  a  deeper  analysis  is 
required.  The  data  in  Table  1  includes  the  entire  F-16  program.  In  order  to  more 
accurately  assess  the  cost  growth  in  the  F-16  program  the  authors  were  able  to  separate 
the  cost  growth  due  to  the  F-16C/D  models  from  that  of  the  F-16A/B  models.  This  is  a 
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more  fair  assessment  of  the  program,  because  one  could  argue  that  at  the  time  of  MS  B 
those  estimating  the  cost  of  the  program  were  not  planning  for  the  F-16C/D  versions. 
Removing  the  F-16C/D  models  from  the  analysis  improves  the  F-16  total  CGF  from  1.42 
to  1 . 1 1 ;  a  significant  decrease  and  a  much  more  accurate  assessment  of  the  cost  growth 
based  on  the  MS  B  estimate.  Table  2  depicts  the  same  basic  data  as  Table  1,  but  this  time 
the  cost  growth  due  to  the  F-16C/D  model  is  removed  for  analysis. 


Table  2:  CGF  Summary  Statistics  with  F-16C/D  Removed 


Category 

Number  of 
Programs 

Mean 

Median 

Standard 

Deviation 

Minimum 

Maximum 

Total 

11 

1.49 

1.31 

0.45 

0.97 

2.40 

RDT&E 

11 

2.05 

1.54 

1.17 

1.32 

5.04 

Procurement 

11 

1.41 

1.36 

0.41 

0.89 

2.03 

Table  3:  CGF  Summary  Statistics  with  F-16C/D  &  UAV  Programs  Removed 


Category 

Number  of 
Programs 

Mean 

Median 

Standard 

Deviation 

Minimum 

Maximum 

Total 

9 

1.32 

1.25 

0.26 

0.97 

1.72 

RDT&E 

9 

1.57 

1.45 

0.33 

1.32 

2.33 

Procurement 

9 

1.27 

1.13 

0.30 

0.89 

1.69 

While  the  results  in  Table  2  are  more  favorable,  the  two  UAV  programs  are  still 


heavily  influencing  the  data.  The  results  in  Table  3  are  much  more  interesting.  Here  the 


total  CGF  due  to  removing  the  two  UAV  programs  has  been  reduced  to  1.32.  Initially 


this  appears  to  be  a  large  improvement,  but  conducting  a  statistical  test  on  the  mean  of 


these  data  results  in  a  P-value  of  0.214  when  compared  to  the  data  from  Arena,  et  al  in 


[8];  meaning  that  there  is  a  21.4%  chance  that  the  null  hypothesis  that  the  mean  for  total 


cost  growth  of  the  prototyped  programs  is  equal  to  the  cost  growth  found  by  Arena,  et  al. 


is  true.  Based  on  these  results  the  null  hypothesis  cannot  be  rejected.  So,  it  would  be 


inaccurate  to  conclude  that  prototyping  definitely  limits  cost  growth  in  all  programs.  The 
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P-value  associated  with  procurement  cost  growth  compared  to  the  research  of  Arena,  et 
al.  was  0.139. 

Expanded  Results 

After  a  closer  look  there  were  four  programs  included  in  both  studies:  the  A- 10, 
AV-8B,  F-16  and  F-18.  Additionally,  Arena,  et  al  included  a  variety  of  weapon  systems 
that  were  not  fixed  wing  aircraft  ranging  from  the  MILSTAR  satellite  to  the  M-1A2 
Abrams  tank.  In  an  effort  to  compare  like  systems  and  to  separate  the  cost  data  from 
programs  that  were  in  both  studies,  SAR  data  was  collected  from  fixed-wing  aircraft  that 
did  not  have  a  prototype  phase  prior  to  FSD.  This  creates  a  data  set  that  can  be  analyzed 
to  compare  cost  growth  specific  to  fixed-wing  aircraft  acquisition  programs  only.  A  total 
of  19  programs  were  used  dating  back  to  1963  (See  Appendix  C  for  complete  list  of 
programs).  The  results  were  captured  and  studied  in  the  same  manner  mentioned  in  the 
previous  chapter.  Table  4  summarizes  the  CGF  categories  for  these  19  non-prototyped 
programs. 


Table  4:  CGF  Summary  for  Non-Prototyped  Fixed  Wing  Aircraft  Programs 


Category 

Number  of 
Programs 

Mean 

Median 

Standard 

Deviation 

Minimum 

Maximum 

Total 

19 

1.36 

1.31 

0.34 

1.00 

1.99 

RDT&E 

19 

1.36 

1.22 

0.39 

0.91 

2.16 

Procurement 

19 

1.33 

1.25 

0.34 

0.96 

2.00 

The  results  in  Table  4,  when  compared  to  Table  2  offer  evidence  that  prototypes 


may  not  have  a  direct  influence  on  a  program’s  cost  growth.  It  is  important  to  note, 


however,  that  many  of  the  programs  selected  in  the  non-prototyped  data  set  were 


programs  that  were  modification  programs.  One  would  expect  that  a  program  based  on 


modifying  a  current  airframe  would  show  significantly  less  cost  growth.  If  these 
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programs  are  removed  from  the  study  then  the  mean  of  the  total  CGF  of  non-prototyped 
programs  goes  up  further  to  1.48  while  procurement  CGF  increased  to  1.52.  A  robust 
increase  from  the  data  depicted  in  Table  4.  However,  based  on  hypothesis  testing,  it 
would  still  be  inaccurate  to  state  that  prototyping  improves  a  program’s  cost  growth 
based  on  the  samples  at  hand. 

Figure  4  offers  a  comparison  of  RDT&E  CGFs  of  the  prototyped  programs,  as 
well  as  the  prototyped  programs  after  removing  the  two  UAV  programs,  against  the 
CGFs  for  the  non-prototyped  programs  and  against  those  same  programs  after  removing 
the  modification  or  derivative  programs.  This  figure  illustrates  that  of  the  original  1 1 
prototype  programs,  CGFs  of  non-U AV  programs  are  much  lower  than  the  prototyped 
program  set  as  a  whole.  The  figure  also  shows  programs  that  were  modifications  or 
derivatives  are  significantly  reducing  the  RDT&E  CGF  of  the  non-prototyped  programs. 


■  Prototypes  □  All  non-prototype 

□  Non-prototype  (no  rrvod/derivative)  TJ  Prototypes  (no  UAV) 
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Figure  4:  Distribution  of  RDT&E  CGFs  of  prototyped  programs,  non-UAV  prototyped  programs, 
non-prototyped  programs  and  non-prototyped  programs  that  are  not  modifications  or  derivatives. 
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Figure  5  shows  the  same  data  for  the  procurement  phase  of  each  acquisition 
program  and  Figure  6Error!  Reference  source  not  found,  does  the  same  for  total 
program  CGF.  Once  again,  the  UAV  programs  are  skewing  the  prototyped  programs 
data  set.  UAV  programs  tend  to  increase  the  average  CGF  of  those  programs  that  used 
early  prototyping,  and  those  programs  considered  to  be  modification  or  derivative 
programs  tend  to  decrease  the  average  CGF  of  the  non-prototyped  programs. 

To  provide  a  comparison  between  like  programs,  another  cost  growth  comparison 
was  made  between  prototyped  programs  that  are  not  UAV  programs  and  are  also  not 
derivative  or  modification  programs  with  non-prototyped  programs  that  were  not 
modification  or  derivative  acquisition  programs.  Table  5  shows  the  summary  statistics 
for  these  two  data  sets.  The  hypothesis  test  for  mean  total  program  CGF  results  in  a  P- 
value  of  0. 159  so  the  difference  is  not  statistically  significant.  However,  a  test  on  the 
mean  procurement  CGF  does  reveal  that  the  means  are  statistically  different  at  nearly  an 
a  =  0.05  level.  So,  it  appears  that  early  prototyping  is  beneficial  for  procurement  cost 
growth  control. 


Table  5:  CGF  Summary  of  Non-Modification/Derivative  Programs  for  Non-UAV  Fixed-Wing 
Aircraft  Programs  that  Used  Early  Prototyping  and  Non-Prototyped  Programs 


Categories 

Number  of 
Programs 

Mean 

Median 

Standard 

Deviation 

Minimum 

Maximum 

Total  (prototyped) 
Total  (non- 

8 

1.36 

1.28 

0.24 

1.19 

1.72 

prototyped) 

6 

1.48 

1.45 

0.32 

1.04 

1.99 

RDT&E  (prototyped) 
RDT&E(non- 

8 

1.60 

1.49 

0.34 

1.36 

2.33 

prototyped) 

Procurement 

6 

1.40 

1.51 

0.30 

0.91 

1.76 

(prototyped) 

Procurement(non- 

8 

1.32 

1.24 

0.28 

1.04 

1.69 

prototyped) 

6 

1.52 

1.53 

0.35 

1.03 

2.00 
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■  Prototypes  □  All  non-prototype 

□  Non-prototype  (no  nrvod/derivative)  TJ  Prototypes  (no  UAV) 
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Figure  5:  Distribution  of  procurement  CGFs  of  prototyped  programs,  non-UAV  prototyped 
programs,  non-prototyped  programs  and  non-prototyped  programs  that  are  not  modifications  or 

derivatives. 
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Figure  6:  Distribution  of  total  program  CGFs  of  prototyped  programs,  non-UAV  prototyped 
programs,  non-prototyped  programs  and  non-prototyped  programs  that  are  not  modifications  or 

derivatives. 
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Last,  the  CGF  for  fighter/attack  aircraft  was  examined  as  these  programs 
comprise  the  majority  of  both  the  early  prototyped  and  non-prototyped  program  data  sets. 
The  summary  statistics  are  shown  in  Table  6  and  Table  7. 

The  mean  of  the  procurement  CGF  for  the  fighter/attack  aircraft  programs  that 
used  early  prototypes  is  statistically  less  than  the  mean  CGF  for  the  non-prototyped  data 
set  at  the  a  =  0. 1  level.  But,  once  again  the  difference  between  the  means  of  the  total 
program  CGF  is  not  statistically  significant. 


Table  6:  CGF  Summary  for  Early  Prototyped  Fighter/ Attack  Aircraft  Programs 


Category 

Number  of 
Programs 

Mean 

Median 

Standard 

Deviation 

Minimum 

Maximum 

Total 

7 

1.33 

1.31 

0.23 

0.97 

1.68 

RDT&E 

7 

1.72 

1.54 

0.49 

1.32 

2.51 

Procurement 

7 

1.28 

1.33 

0.28 

0.89 

1.69 

Table  7:  CGF  Summary  for  Non-Prototyped  Fighter/Attack  Aircraft  Programs 


Category 

Number  of 
Programs 

Mean 

Median 

Standard 

Deviation 

Minimum 

Maximum 

Total 

8 

1.46 

1.35 

0.34 

1.03 

1.98 

RDT&E 

8 

1.47 

1.51 

0.44 

0.98 

2.16 

Procurement 

8 

1.44 

1.37 

0.30 

1.06 

1.90 

The  analysis  shows  that  it  would  be  inaccurate  to  assume  that  prototyping  ensures 
a  more  accurate  cost  estimate  across  all  acquisition  programs.  Prototyping  does  not 
appear  to  improve  cost  growth  in  either  the  RDT&E  phase  or  for  the  total  program. 
Prototyping  does,  however,  appear  show  benefits  in  the  procurement  phase;  giving 
programs  with  large  procurement  buys  the  opportunity  for  a  return  on  investment. 
Program  directors  with  an  acquisition  program  expecting  to  purchase  a  high  number  of 
units,  should  strongly  consider  prototyping  as  a  possible  strategy. 
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Relationships  Between  CGF  and  Other  Program  Characteristics 
Cost  growth  is  not  the  only  important  aspect  to  analyzing  the  overall  success  of  a 
program.  Another  important  measure  of  effectiveness  is  schedule  growth.  Studies  have 
shown  that  longer  programs  tend  to  have  higher  cost  growth  [8],  If  this  were  accurate 
then  programs  with  schedule  slip  would  show  extensive  cost  growth,  but  Figure  7  shows 
no  discernible  relationship  between  schedule  growth  to  initial  operational  capability 
(IOC)  and  CGF  for  the  fixed-wing  aircraft  programs  that  used  prototypes  prior  to  MS  II 
or  MS  B.  The  Pearson  correlation  coefficient  revealed  very  weak  linear  dependence 
between  schedule  growth  factor  (SGF)  and  CGF.  Figure  8  also  reveals  no  relationship 
between  the  CGF  and  the  number  of  months  that  actually  transpire  between  MS  II  or  B 
and  IOC  for  the  set  of  programs  that  used  early  prototyping. 
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Figure  7:  Relationship  of  a  program’s  SGF  to  its  CGF 
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ORDT&E  □  Procurement  ATotal 


Months  to  IOC 


Figure  8:  Relationship  of  a  program’s  length  of  schedule  to  IOC  to  its  CGF 

The  next  relationship  examined  was  between  CGF  and  the  proportion  of  program 
costs  that  were  spent  during  the  early  prototyping  phase.  The  proportion  of  program’s 
cost  was  calculated  by  first  removing  the  cost  of  the  prototype  phase  from  the  RDT&E 
cost.  Then  the  prototype  phase  cost  was  divided  by  the  remaining  RDT&E  cost  and  by 
the  remaining  total  program  cost.  Figure  9  displays  the  percentage  of  RDT&E  spent  on 
the  prototype  phase  and  relates  it  to  the  program’s  RDT&E,  procurement  and  total  CGF. 
Figure  10  portrays  the  prototype  cost  as  a  percentage  of  the  entire  program’s  cost  and 
relates  this  to  the  individual  CGFs.  In  both  cases,  no  discernible  relationship  exists  and 
once  again  the  Pearson  correlation  coefficient  reveals  only  weak  linear  dependence. 
Contrary  to  what  one  might  expect,  programs  that  spent  a  small  proportion  of  funds  on 
early  prototyping  seem  to  fair  just  as  well  as  programs  that  spend  more  on  the  early 
prototyping  phase. 
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<>  RDT&E  □  Procurement  ATotal 


Prototype  Cost  as  Poroportion  of  Full  Scale  Development  RDT&E 


Figure  9:  CGF  related  to  the  proportion  of  RDT&E  funds  dedicated  to  early  prototyping 
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Summary 

There  is  little  evidence  that  on  average  early  prototyping  is  any  more  effective  at 
controlling  cost  growth  than  other  acquisition  strategies.  The  only  caveat  to  this 
statement  is  when  UAV  programs  are  not  included  in  the  discussion  then  early 
prototyping  does  appear  to  lead  to  lower  procurement  and  total  cost  growth  for  the 
follow-on  acquisition  program.  The  data  were  also  analyzed  in  the  hopes  of  finding 
relationships  between  cost  growth  and  other  program  characteristics.  However,  no 
relationship  between  CGF  and  SGF,  schedule  length  or  the  proportion  of  program  costs 
expended  during  the  early  prototyping  phase  exist. 
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V.  Conclusions  and  Recommendations 


Conclusions  of  Research 

Early  prototyping  is  not  a  cure-all  that  guarantees  success  in  acquisition  with 
respect  to  cost  growth.  Early  prototyping  was,  however,  used  for  several  successful 
fixed-wing  aircraft  acquisition  programs.  For  these  programs,  prototyping  has  been 
counted  among  the  reasons  for  the  success  of  the  acquisition  program  that  followed. 
And,  although  the  data  did  not  present  any  statistically  significant  relationships  for  total 
program  cost  growth  or  rules  of  thumb  for  the  use  of  prototypes  as  part  of  fixed-wing 
aircraft  acquisition  programs,  there  are  still  some  insights  that  can  be  drawn  from  this 
research. 

The  prototype  program  structure  or  management  style  appears  to  have  a  strong 
bearing  on  the  success  of  the  acquisition  program  that  follows  just  as  an  acquisition 
program’s  management  structure  is  a  determinant  of  performance  [23],  The  prototype 
programs  executed  during  the  1970s  (Close  Air  Support  (A-X),  Light  Weight  Fighter 
(LWF),  Advanced  Medium  STOL  Transport  (AMST)  and  HAVE  BLUE)  had  many 
similarities  in  program  structure  as  described  in  their  respective  case  studies  [40]  [41] 
[32]  [31].  With  the  exception  of  the  C-17A,  all  the  acquisition  programs  that  followed 
these  prototyping  programs  experienced  lower  than  average  cost  growth.  By  most 
accounts  these  prototype  programs  led  to  acquisition  programs  that  were  successful  in 
delivering  the  perfonnance  required,  and  delivering  an  adaptable  weapon  system  that 
went  on  to  deliver  performance  well  beyond  what  was  envisioned  at  program  initiation. 
The  Advanced  Tactical  Fighter  (ATF)  program  also  used  a  similar  prototyping  program 
structure  that  was  modeled  after  the  lessons  learned  from  these  predecessors  [42].  As 
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mentioned  previously,  most  of  the  acquisition  programs  that  followed  from  these 
prototype  programs  perfonned  well  in  tenns  of  total  acquisition  program  cost  growth. 
This  was  primarily  due  to  relatively  low  procurement  cost  growth. 

Regarding  the  C-17,  a  case  study  of  the  acquisition  program  would  be  required  to 
uncover  the  reasons  for  the  higher  than  average  cost  growth.  However,  a  few 
observations  will  show  that  the  acquisition  program  did  experience  external  factors  that 
likely  exerted  a  strong  influence  on  the  cost  growth.  One  significant  difference  between 
the  C-17  program  and  the  programs  that  followed  the  other  prototype  efforts  is  the  delay 
between  the  prototyping  and  acquisition  programs.  The  AMST  prototype  program 
officially  ended  in  1979  (the  test  program  was  complete  by  early  1978),  but  the  C-17 
program  did  not  start  until  1985.  The  lengthy  delay  between  prototype  phase  and 
acquisition  program  is  not  characteristic  of  the  other  programs.  Also,  in  addition  to 
underestimating  the  resources  required  for  the  program,  the  C-17  program  experienced 
changes  external  to  the  program  office  and  contractor  in  the  form  of  requirements, 
funding  and  production  stretch  [43]  [44], 

Moving  back  to  the  topic  of  procurement  cost  growth,  the  other  programs 
mentioned  above  showed  lower  than  average  growth  (compared  to  the  RAND  study  and 
the  fixed-wing  aircraft  comparison  samples)  except  for  the  F-18.  However,  most  of  these 
programs  had  higher  than  average  RDT&E  cost  growth  compared  to  the  fixed-wing 
aircraft  comparison  sample.  The  lesson  learned  is  that  prototyping  at  the  system  level 
may  be  best  applied  to  programs  that  expect  a  large  production  run. 

The  previously  mentioned  case  studies  provide  some  of  the  common  features  of 
these  prototype  programs  that  were  described  as  contributors  to  success.  Note  that  these 
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are  characteristics  of  the  prototype  programs,  not  necessarily  the  acquisition  programs 
that  followed.  Program  offices  that  are  considering  a  prototype  program  prior  to  MS  B 
would  do  well  to  examine  the  costs  and  benefits  of  these  program  features. 

1 .  Small  and  empowered  program  offices  ( 1  Os  of  people) 

2.  Minimum  documentation  requirements 

3.  Issued  clear  perfonnance  goals  rather  than  specifications  for  compliance. 

4.  Contractors  were  allowed  to  make  tradeoffs  to  provide  best  overall 
performance  and  demonstrate  the  technologies  of  interest 

5.  Contractors  were  allowed  to  create  the  schedule  but  were  held  accountable 
once  the  clock  started 

6.  Prototypes  were  relevant  to  the  follow-on  acquisition  programs  in  terms  of 
risks  addressed  and  the  technologies  that  were  included  in  the  initial 
acquisition  program  that  followed 

7.  Prototypes  were  relatively  low  cost  and  austere  (only  the  A- 10  and  F-22 
exceeded  a  prototype  program  cost  of  10%  of  the  acquisition  program 
RDT&E) 

8.  Program  office  personnel  maintained  close  communication  (technical  and 
programmatic)  with  the  contractors,  which  provided  insight  rather  than 
oversight 

9.  Testing  was  targeted  to  show  evidence  of  meeting  overall  performance  goals 
and  the  test  teams  featured  combined  developmental  and  operational  testers 
from  the  government  along  with  contractor  test  personnel.  The  test  teams 
were  also  empowered  to  make  in-scope  changes  to  the  test  plan. 
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10.  Contract  structures  allowed  the  contractors  to  take  risks  while  protecting  the 
government  from  cost  overruns  by  using  cost  ceilings  or  fixed-price  contracts 
(this  falls  in-line  with  the  concept  that  some  prototypes  will  fail,  but  measured 
risk  taking  should  not  be  avoided) 

1 1 .  The  programs  offered  legitimate  competition  with  no  clear  front  runner  at  the 
beginning  of  the  program  and  a  significant  prize  to  be  won  for  providing  the 
best  overall  design 

Additionally,  some  of  these  programs  used  extensive  subsystem  prototyping 
and/or  modeling  and  simulation  to  support  the  full  scale  aircraft  prototyping  program. 
Two  examples  of  this  are  the  A-X  and  ATF  programs.  The  A-X  program  prototyped  the 
gun  system  and  ammunition  while  ATF  pursued  extensive  avionics  and  engine 
development.  These  efforts  were  external  to  the  competitive  fly-off  prototype  programs. 
In  both  cases,  the  subsystem  technologies  were  believed  to  be  as  high  or  higher  risk  than 
the  technologies  demonstrated  on  the  aircraft  prototypes. 

The  fixed-wing  aircraft  programs  that  have  used  prototyping  most  recently  are 
showing  poorer  cost  growth  perfonnance.  Two  of  the  three  programs  that  kicked-off 
their  prototyping  efforts  in  the  1990s  were  UAV  programs;  the  third  was  F-35.  Both  of 
these  UAV  programs  used  advanced  concept  technology  demonstration  (ACTD) 
programs.  The  data  show  that  these  programs  had  much  higher  than  average  cost  growth, 
but  the  actual  schedule  from  MS  B  to  the  end  of  initial  operational  test  and  evaluation 
(IOT&E)  and  to  initial  operational  capability  (IOC)  declaration  were  relatively  short 
compared  to  the  other  prototyped  programs.  So,  this  could  be  an  indication  that  using  an 
ACTD  allows  programs  to  buy  a  shortened  schedule  for  UAVs.  This  conclusion  is 
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uncertain,  however,  since  these  also  happen  to  be  the  only  UAV  programs  in  the  study, 
and  the  only  other  ACTD  program  in  the  study  is  F-l  17.  In  the  case  of  F-35,  [16]  and 
[45]  provide  several  reasons  that  cost  growth  occurred. 

Two  of  the  documented  root  causes  for  the  high  cost  growth  of  F-35  apply 
particularly  well  to  prototyping.  First,  competing  design  requirements  among  the  three 
variants  of  the  aircraft  were  not  discovered  or  sufficiently  highlighted  by  the  prototyping 
phase.  One  primary  purpose  of  early  prototypes  is  to  prove  the  feasibility  and  utility  of 
requirements  and  operational  concepts,  but  the  program  must  design  test  objectives  that 
provide  for  collection  of  these  data.  Second,  the  demonstrators  did  not  make  use  of  key 
technologies  to  demonstrate  capabilities.  Early  prototyping  must  be  focused  on 
demonstrating  technology  maturity  relevant  to  achieving  required  capabilities  on  the 
follow-on  weapon  system.  Prototyping  the  wrong  technology  will  not  contribute  to  good 
decision  making  with  regard  to  committing  to  a  weapon  system  and  an  accompanying 
acquisition  strategy.  Prototyping  the  wrong  technology  implies  any  of  the  following: 

1 .  Technology  incorporated  on  the  follow-on  weapon  system  is  different  or 
highly  modified  from  what  is  demonstrated. 

2.  Avoiding  high  risk  technology  areas  on  the  prototype. 

3.  Adding  new  high  risk  technologies  after  completion  of  the  prototyping  phase. 

4.  Ignoring  integration  of  immature  technologies  (not  addressing  integration 
immaturity). 

In  summary,  for  F-35  the  “concept  demonstrator  may  not  have  been  sufficient 
preparation  for  executing  SDD  [16].” 
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These  thoughts  lead  to  some  remarks  on  acquisition  policy.  The  WSARA  2009 
makes  competitive  prototyping  an  expectation  for  MDAPs.  The  data  in  this  study  do  not 
support  such  a  strong  mandate  on  the  use  of  prototyping.  Rather  than  forcing  a  program 
to  pursue  a  waiver  to  avoid  prototyping,  programs  should  simply  be  required  to  examine 
the  costs  and  benefits  of  prototyping  prior  to  MS  B.  WSARA  rightly  points  out  that 
prototyping  does  not  necessarily  mean  system  level  prototypes,  but  may  be  subsystem 
prototypes  only.  The  cost-benefit  analysis  should  be  the  required  deliverable  to  the  MDA 
at  MS  A.  The  difference  in  these  approaches  is  the  point  of  emphasis  and  the  implied 
level  of  overhead  for  programs.  Acquisition  policy  needs  to  emphasize  accomplishing 
the  analysis,  which  is  part  of  sound  systems  engineering,  rather  than  the  paperwork  trail. 
Unfortunately,  the  misplaced  emphasis  in  WSARA  was  preceded  by  poor  policy  in  DoDI 
5000.02.  As  was  quoted  in  Chapter  II,  DoDI  5000.02  clearly  states  that  competitive 
prototyping  will  be  used.  Rather,  the  policy  should  require  performance  of  a  cost-benefit 
analysis  of  prototyping,  state  where  findings  will  be  documented,  and  describe  example 
objectives  of  prototyping  and  prototyping  strategies  that  program  offices  could  consider. 
If  a  prototyping  program  is  not  designed  specifically  to  meet  objectives  for  a  particular 
program  it  may  well  result  in  spending  extra  time  and  money  up  front  with  no 
recognizable  benefits  to  the  program  later  in  the  life  cycle.  Therefore,  program  managers 
need  to  target  the  prototyping  to  aspects  of  the  program  where  knowledge  is  lacking  most 
or  risk  is  highest.  An  austere  prototype  with  a  properly  focused  test  plan  likely  provides 
greater  cost-benefit  than  a  catch  all  prototype. 

One  other  conclusion  that  can  be  drawn  from  the  data  presented  is  that,  fixed- 
wing  aircraft  modification  programs  experience  lower  cost  growth  compared  to  new  start 
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programs.  This  should  not  be  an  earth  shattering  conclusion.  The  good  cost  growth 
perfonnance  of  modification  programs  was  previously  noted  by  Drezner,  et  al.  in  [13]. 
The  F-18E/F  model  is  an  excellent  example.  The  modification  of  the  F-18  from  its 
original  concept  up  to  the  E/F  models  has  resulted  in  significantly  improved  product,  but 
has  retained  its  same  basic  design.  This  acquisition  strategy  resulted  in  a  procurement 
CGF  of  1 .06  and  total  program  CGF  of  1 .03 !  These  results  are  vastly  superior  to  similar 
new  start  programs  during  the  same  timeframe.  The  fact  that  these  programs  perform  so 
much  better  than  new  start  programs  is  not  surprising;  however,  the  benefits  of  pursuing  a 
modification  program  may  go  underappreciated  at  times.  Unless  a  weapon  system  is 
shown  to  be  unsuitable  for  an  emerging  threat  or  mission  area  because  of  an  inherent 
technology  limitation  (e.g.,  lack  of  low  observable  technology)  then  the  services  should 
strongly  consider  a  modification  program  to  fill  new  requirements  and  reserve  new  start 
programs  for  revolutionary  or  ‘leap  ahead’  technologies.  The  F- 117  and  F-22  provide 
examples  of  revolutionary  aircraft.  In  this  case,  obtaining  a  functional  revolutionary 
technology  application  justifies  the  resources  expended  and  the  uncertainty  of  resource 
commitment  (i.e.,  risk).  A  revolutionary  technology  application  program  provides  the 
perfect  example  for  extensive  early  prototyping. 

The  final  lesson  learned  that  this  research  identified  is,  for  program  managers  and 
MDAs  to  review  past,  like  programs  to  examine  acquisition  strategies  that  would  be 
applicable  to  a  new  program.  Although  DoD  5000  provides  an  outline  of  required 
documentation  to  navigate  the  acquisition  process,  it  is  not  designed  (nor  should  it  be)  to 
provide  a  one  stop  shop  of  best  practices  that  are  most  applicable  to  every  type  of 
acquisition  program.  Certainly  the  intent  of  DoD  5000  is  to  force  programs  down  the 
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path  of  best  practices  via  the  documentation  requirements,  however,  this  should  not 
replace  some  independent  study  on  past  programs  to  uncover  lessons  learned.  There  are 
many  pieces  to  an  acquisition  strategy  (early  prototyping  being  one)  that  should  be 
examined  for  applicability  as  any  new  program  is  established.  Discovering  what  worked, 
and  possibly  more  importantly,  the  root  causes  of  what  did  not  work  in  past  acquisitions 
might  be  one  of  the  most  important  steps  to  establishing  a  successful  acquisition  program. 
MDAs  would  be  wise  to  set  the  expectation  that  program  managers  show  evidence  of 
their  technology  maturity  and  integrated  early  RDT&E  as  they  present  an  acquisition 
strategy  for  approval. 

Significance  of  Research 

This  research  shows  that  early  prototyping  may  not  be  appropriate  for  all 
programs  or  it  can  be  misapplied,  resulting  in  prototyped  programs  that  perform  no 
better,  with  respect  to  cost  growth,  than  programs  that  did  not  use  early  prototyping. 
Prototyping  is  not  a  panacea,  and  must  be  carefully  planned  and  executed  using  lessons 
learned  from  past  programs  that  implemented  early  prototyping  successfully. 

Recommendations  for  Action 

Rewrite  DoDI  5000.02  to  emphasize  the  cost-benefit  analysis  of  prototyping 
strategies  and  objectives.  Program  managers  need  to  ask  the  questions,  “How  can 
prototyping  help  me  make  a  better  decision  at  MS  B,  and  is  the  value  of  that  knowledge 
worth  the  cost?” 
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Recommendations  for  Future  Research 


As  this  study  progressed  a  suspicion  arose  that  there  may  well  be  more  to  the 
story.  A  more  complete  description  of  the  costs  and  benefits  of  early  prototyping  would 
be  possible  if  completed  weapon  system  performance  were  quantitatively  described  with 
respect  to  the  perfonnance  planned  at  MS  II  or  MS  B.  Additionally,  tracking  the 
progress  of  technology  readiness  levels  throughout  the  prototyping  phase  provides 
another  measure  of  effectiveness  for  early  prototyping.  Research  of  this  nature  would 
likely  require  more  of  a  case  study  approach  to  each  program  examined.  Finally,  a  case 
study  approach  into  these  prototyped  programs  is  appropriate  to  reveal  other  program 
characteristics  that  contributed  to  cost  growth  (i.e.,  discovering  root  causes). 

Secondly,  the  benefits  of  pursuing  modification  programs  rather  than  new  start 
programs  could  pose  a  profitable  investigation.  To  be  complete  this  would  also  require  a 
study  of  the  technologies  along  with  the  costs  and  schedule.  As  mentioned  previously, 
the  authors  suspect  that  modification  programs  should  be  the  first  alternative  to  a  material 
solution  unless  a  prohibitive  feature  of  the  emerging  need  exists  (e.g.,  the  problem  posed 
by  anti-access  threats  to  non-LO  aircraft).  An  example  of  this  would  be  to  perform  a 
cost-benefit  analysis  of  modifying  the  F-22  rather  than  pursuing  the  development  of  the 
F-35  might  reveal  interesting  results. 

Summary 

By  using  SAR  data  and  historical  program  documents  the  authors  were  able  to 
characterize  the  quantity  and  inflation  adjusted  cost  growth  for  fixed-wing  aircraft 
programs  that  used  prototyping  prior  to  MS  II  or  MS  B.  The  data  show  that,  on  the 
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whole,  prototyping  did  not  lead  to  lower  average  cost  growth.  However,  when  UAV 
programs  are  removed  from  the  data  set,  the  average  cost  growth  is  lower  than  the 
average  cost  growth  of  the  comparison  sample  sets.  In  this  case,  the  difference  was  only 
statistically  significant  for  procurement  cost  growth  when  compared  to  the  fixed-wing 
aircraft  programs  that  did  not  use  early  prototyping  and  were  not  modification  or 
derivative  programs.  No  relationship  could  be  identified  between  the  cost  growth  of 
these  prototyping  programs  and  proportional  cost  of  the  prototyping  phase,  schedule 
growth,  or  length  of  schedule.  Two  of  eleven  programs  examined  caused  a  significant 
increase  in  the  overall  average  cost  growth  for  the  entire  sample. 

The  language  used  in  WSARA  2009  and  the  prototyping  policy  in  DoDI  5000.02 
are  well  intentioned  but  poorly  written.  Rather  than  encouraging  programs  to  examine 
how  early  prototyping  can  benefit  a  program  and  to  estimate  the  costs  of  these  benefits, 
the  policy  simply  says  prototyping  will  be  done  in  the  Technology  Development  phase. 
WSARA  does  state  that  the  prototyping  requirement  can  be  waived,  however,  pursuing  a 
waiver  is  just  another  piece  of  unnecessary  bureaucratic  administration.  Defense 
acquisition  policy  needs  to  emphasize  how  to  use  early  prototyping  to  gain  knowledge  for 
making  better  decisions  at  MS  B.  MDAs  should  hold  program  managers  accountable  at 
the  decision  briefing  by  asking  why  particular  aspects  of  an  acquisition  strategy  were 
chosen,  instead  of  using  a  waiver  to  show  accountability. 
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Appendix  A  -  Acronyms 


AMST 

Advanced  Medium  STOL  Transport 

Cost  Growth  Factor 

CIC 

Cost  Improvement  Curve 

DoD 

Department  of  Defense 

EMD 

Engineering,  Manufacturing  and  Development 

FSD 

Full  Scale  Development 

IDA 

Institute  for  Defense  Analyses 

IOC 

Initial  Operational  Capability 

MDA 

Milestone  Decision  Authority 

MDAP 

Major  Defense  Acquisition  Program 

MS 

Milestone 

PDR 

Preliminary  Design  Review 

SAR 

Selected  Acquisition  Report 

SDD 

System  Development  and  Demonstration 

SGF 

Schedule  Growth  Factor 

STOL 

Short  Takeoff  and  Landing 

UAV 

Unmanned  Aerial  Vehicle 

WSARA 

Weapon  System  Acquisition  Reform  Act 
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Appendix  B  -  Prototyped  Programs  Included  in  Study 


Table  8:  Prototype  Programs  Included  in  this  Study 


Weapon 

System 

Prototype 

Aircraft  Type 

ACTD 

Derivative 

Base 

Year 

Quantity  Norm 
Method 

A-10 

YA-10 

Fighter/ Attack 

No 

No 

1970 

SAR  variances 

AV-8B 

YAV-8B 

Fighter/ Attack 

No 

Yes 

1979 

CIC 

C-17A 

C-15 

Airlift 

No 

No 

1996 

CIC 

F-16 

YF-16 

Fighter/ Attack 

No 

No 

1997 

CIC 

F-18A/B/C/D 

YF-17 

Fighter/ Attack 

No 

No 

1975 

CIC 

F-22 

YF-22 

Fighter/ Attack 

No 

No 

2005 

CIC 

F-35 

X-35 

Fighter/ Attack 

No 

No 

2012 

CIC 

F-117 

HAVE  BLUE 

Fighter/ Attack 

Yes 

No 

2010 

CIC 

MQ-1 

Predator 

UAV 

Yes 

No 

2008 

CIC 

RQ-4 

Global  Hawk 

UAV 

Yes 

No 

2000 

CIC 

V-22 

XV- 15 

Airlift 

No 

No 

2005 

CIC 
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Appendix  C  -  Non-Prototyped  Programs  Included  in  Study 


Table  9:  Fixed-Wing  Aircraft  Non-Prototyped  Programs  Included  in  the  Study 


Weapon  System 

Aircraft  Type 

Derivative 

Base  Year 

Quantity  Norm 
Method 

F-18E/F 

Fighter/ Attack 

Yes 

2000 

CIC 

F-15 

Fighter/ Attack 

No 

1970 

CIC 

JPATS  (T-6  II) 

Trainer 

Yes 

2002 

CIC 

T-45 

Trainer 

No 

1995 

CIC 

P-8 

ISR 

Yes 

2010 

CIC 

MQ-4C 

UAV 

Yes 

2008 

CIC 

B-1B 

Bomber 

Yes 

1981 

CIC 

F-l  1 1A/C/D/E/F 

Fighter/ Attack 

No 

1963 

CIC 

EF-111A 

Fighter/ Attack 

Yes 

1973 

SAR  variance 

E-6A 

C2 

Yes 

1982 

CIC 

F-14A 

Fighter/ Attack 

No 

1969 

CIC 

F-14D 

Fighter/ Attack 

Yes 

1989 

CIC 

KC-10A 

Tanker 

Yes 

1976 

CIC 

JSTARS 

C2 

Yes 

1998 

CIC 

S-3A 

C2 

No 

1968 

SAR  variance 

C-5A 

Airlift 

No 

ty* 

SAR  variance 

A-7D 

Fighter/ Attack 

Yes 

1967 

SAR  variance 

A-7E 

Fighter/ Attack 

Yes 

1967 

SAR  variance 

E-3A 

C2 

Yes 

1970 

CIC 

*The  C-5  SARs  did  not  provide  base-year  data.  The  then-year  cost  estimate  was  for  the  years  FY 65  to 
FY71  while  the  final  estimate  was  for  FY65  to  FY74  (the  FY74  budget  was  less  than  1%  of  the  total 
program  budget).  Since  the  data  are  not  presented  in  the  form  of  annual  costs  on  the  C-5  SAR  the  data 
could  not  be  reliably  inflated  to  a  base-year.  Therefore  the  then-year  data  were  used,  which  overestimates 
the  cost  growth  slightly. 
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